Hey Alex, the PR is looking super solid, thanks again for all the effort you put in!
I'm wondering if you could also add the following tests:

1. If you drop a table with an existing order, and then create a table with the same name but different columns?
2. If you get the changes information on snapshot()
3. If the alter statement won't intefeer with compaction (see, Change schema tracking from global to table. #671)
4. If we can also verify the new parquet files are correctly written directly (without acessing them from the table), and also if different files have the correct orders, when having different orders over multiple snapshots.
5. More transaction tests, for example, what happens if we begin a transaction and in the same transaction we add a new sorting order and alter a column name?
6. What happens if you do a batch insertion on a table with a set column order (not inlined)

The test files you are pretty detailed, but my bio llm context window is small, do you mind also moving them to a sorted_table folder and modularizing them a bit more?
I also guess in general we can do StringUtil::Lower() for catalog access when the result can be case insensitive.

Thanks as always for the feedback! These tests make sense and I'm working on them! I had a few follow up questions.

On 3, would you like for me to validate that an alter table statement on a different table does not interfere? (Ahead of time or in the same transaction?) Based on #684, I was assuming that any alter table on the same table would be a hard line that prevents compaction - please let me know if I am wrong!

Basically, we should check that the alter statement does not increase the schema counter of a table, since the alter statement here does not change the actual schema, and the sorting is somewhat "optional" I believe that files should still be compactable before/after the alter.

On 4, let me try to repeat back to you what I think you are looking for! I think you want to test the situation where I insert into the same table 3 times, each time with a different sort order setting, and then I run 1 compaction on that table. My goal was to only consider the most recent ducklake_sort_expression setting, regardless of the value of the setting at insert time, so this test would ensure that! I could also test multiple compactions and make sure that the final compaction resorts to the latest sort setting.

I wanted to see if files would always be generated with the latest sorting (e.g., by flushing) , and also compacted with the latest sorting, but now that I think about it again, maybe is an overkill because you already have similar tests to that, let's skip it.

On 6, I'm not sure exactly how to proceed - I could use your help! This PR does not include the capability to do the sorting during an insert, so I'm not sure how the features would intersect just yet. Does adjusting the column order during insert result in the parquet file having the columns stored in a different order? I'm not sure what behavior I should be exercising with the test. Just let me know!

Let's skip this.

When investigating 3, I found that you are correct that I am currently incrementing the schema_version when altering the sort order. Dang! I think there are few other cases where we probably want to have the same "alter table, but don't change the schema_version". I'm thinking:

• SET_SORT_KEY
• SET_COMMENT
• SET_COLUMN_COMMENT

I'll start trying to implement that behavior, but let me know if I need to course correct! From my first prototype, it requires some additional complexity in a few spots (I think we will need a different construct than new_tables for storing those kinds of alters), so I'm open to feedback and ideas!

When investigating 3, I found that you are correct that I am currently incrementing the schema_version when altering the sort order. Dang! I think there are few other cases where we probably want to have the same "alter table, but don't change the schema_version". I'm thinking:

• SET_SORT_KEY
• SET_COMMENT
• SET_COLUMN_COMMENT

I'll start trying to implement that behavior, but let me know if I need to course correct! From my first prototype, it requires some additional complexity in a few spots (I think we will need a different construct than new_tables for storing those kinds of alters), so I'm open to feedback and ideas!

Yes, I think none of those should impact the schema ranges for compaction. I think the main thing you have to do is to ensure that these statements won't write a new entry into ducklake_schema_versions, as this table keeps track of schema changes per ducklake table.

Yup, I will prevent that! Unfortunately, I'm also finding that I need to update the local schema cache (since usually a new schema version would be handled differently). I'm making progress!

I now have the schema_version not incrementing on SET_SORT_KEY, SET_COMMENT, and SET_COLUMN_COMMENT!

I split the tests apart and also added tests for 1, 2, 3, and 5! I added quite a lot of tests to ensure transactionality for 5. Since alter table can't happen in the same transaction as a compaction (Getting the deliberate Transactions can either make changes OR perform compaction - not both error), that meant fewer cases in the compaction case than the inlining.

I also added tests for the schema_version non-incrementing for the comment cases.

Let me know what you think!

So, to fix the assertion issues on my fork's CI, I had to relax an assertion. Please let me know if I am off base, but I think that the assertion was too tight.

In src/storage/ducklake_catalog.cpp lines 439-461, the table_entry_map can have views added to it. However, there was an assertion in DuckLakeCatalogSet::GetEntryById(TableIndex index) that required a table (and not a view). Allowing a view there appears to solve things.

As I am thinking more about this, I'm wondering if the updates I made to the catalog cache would be safe across processes. Is it safe to not update the schema_version? Could other processes use a stale sort order, or reset it back to being unsorted?

So, to fix the assertion issues on my fork's CI, I had to relax an assertion. Please let me know if I am off base, but I think that the assertion was too tight.

In src/storage/ducklake_catalog.cpp lines 439-461, the table_entry_map can have views added to it. However, there was an assertion in DuckLakeCatalogSet::GetEntryById(TableIndex index) that required a table (and not a view). Allowing a view there appears to solve things.

Could you point me out to which test broke this requirement?

From what I can tell, there are other parts of the code that require this to return a table, as we deference it to a DuckLakeTableEntry

e.g.,

unique_ptr<DuckLakeStats> DuckLakeCatalog::ConstructStatsMap(vector<DuckLakeGlobalStatsInfo> &global_stats,
                                                             DuckLakeCatalogSet &schema) {
	auto lake_stats = make_uniq<DuckLakeStats>();
	for (auto &stats : global_stats) {
		// find the referenced table entry
		auto table_entry = schema.GetEntryById(stats.table_id);

As I am thinking more about this, I'm wondering if the updates I made to the catalog cache would be safe across processes. Is it safe to not update the schema_version? Could other processes use a stale sort order, or reset it back to being unsorted?

Maybe this is something we can test with concurrentloops? (see test/sql/snapshot_info/ducklake_last_commit.test)

So, to fix the assertion issues on my fork's CI, I had to relax an assertion. Please let me know if I am off base, but I think that the assertion was too tight.
In src/storage/ducklake_catalog.cpp lines 439-461, the table_entry_map can have views added to it. However, there was an assertion in DuckLakeCatalogSet::GetEntryById(TableIndex index) that required a table (and not a view). Allowing a view there appears to solve things.

Could you point me out to which test broke this requirement?

From what I can tell, there are other parts of the code that require this to return a table, as we deference it to a DuckLakeTableEntry

e.g.,

unique_ptr<DuckLakeStats> DuckLakeCatalog::ConstructStatsMap(vector<DuckLakeGlobalStatsInfo> &global_stats,
                                                             DuckLakeCatalogSet &schema) {
	auto lake_stats = make_uniq<DuckLakeStats>();
	for (auto &stats : global_stats) {
		// find the referenced table entry
		auto table_entry = schema.GetEntryById(stats.table_id);

Sure! The tests that broke are here in this CI run on my fork. They were all running a query like

COMMENT ON VIEW ducklake.comment_view IS 'con1';

I can create a view-specific GetEntryById function if that would be better!

As I am thinking more about this, I'm wondering if the updates I made to the catalog cache would be safe across processes. Is it safe to not update the schema_version? Could other processes use a stale sort order, or reset it back to being unsorted?

Maybe this is something we can test with concurrentloops? (see test/sql/snapshot_info/ducklake_last_commit.test)

Unfortunately, I believe that concurrentloop uses the same DuckDB instance with multiple threads, and the DuckLake catalog is only created once per ATTACH, so it is shared across all threads. I think the issue that might exist from no longer incrementing schema_version would be when two totally separate DuckLakeCatalog instances have different sort information in their cache (with the same schema_version). Is there a multi-process version of concurrentloop? Or maybe a C++ or Python test? Do you want me to remove the schema_version modifications and save them for a later PR?

As I am thinking more about this, I'm wondering if the updates I made to the catalog cache would be safe across processes. Is it safe to not update the schema_version? Could other processes use a stale sort order, or reset it back to being unsorted?

Maybe this is something we can test with concurrentloops? (see test/sql/snapshot_info/ducklake_last_commit.test)

Unfortunately, I believe that concurrentloop uses the same DuckDB instance with multiple threads, and the DuckLake catalog is only created once per ATTACH, so it is shared across all threads. I think the issue that might exist from no longer incrementing schema_version would be when two totally separate DuckLakeCatalog instances have different sort information in their cache (with the same schema_version). Is there a multi-process version of concurrentloop? Or maybe a C++ or Python test? Do you want me to remove the schema_version modifications and save them for a later PR?

Could we achieve this with multiple connections then? Because that's also possible within sqltests

As I am thinking more about this, I'm wondering if the updates I made to the catalog cache would be safe across processes. Is it safe to not update the schema_version? Could other processes use a stale sort order, or reset it back to being unsorted?

Maybe this is something we can test with concurrentloops? (see test/sql/snapshot_info/ducklake_last_commit.test)

Unfortunately, I believe that concurrentloop uses the same DuckDB instance with multiple threads, and the DuckLake catalog is only created once per ATTACH, so it is shared across all threads. I think the issue that might exist from no longer incrementing schema_version would be when two totally separate DuckLakeCatalog instances have different sort information in their cache (with the same schema_version). Is there a multi-process version of concurrentloop? Or maybe a C++ or Python test? Do you want me to remove the schema_version modifications and save them for a later PR?

Could we achieve this with multiple connections then? Because that's also possible within sqltests

I am not sure! If you have a spot where I can find an example, I can give it a shot.

To understand the behavior, I made a Python script that kicks off 2 separate CLI processes. I found that the sort is ignored by the other process if the schema was already cached ahead of time. The good news is that the catalog DB itself continues to have the right values, but the cache does not get invalidated correctly.

The flow is:

• Process 1: connects, creates the table and inserts into it
• Process 2: connects and runs an ALTER TABLE ADD COLUMN, which caches the catalog
• Process 1: ALTER TABLE SET SORTED BY
• Process 1: Completes / exits
• Process 2: Compacts (using the cached catalog)
• Process 2: Pulls updated table (which does not show the right order, since the cached catalog was used)
• Process 2: Completes / exits

If I omit the ALTER TABLE ADD COLUMN step in process 2, then there is no issue and the sort occurs correctly.

What do you recommend I do? I've thought about 3 options, but 3 would need some help!

1. Keep the schema_version from incrementing, but accept this concurrency behavior.
2. Allow the schema_version to increment but accept that a compaction barrier gets put in when sort is changed
3. Keep the schema_version from incrementing but find some other way to correctly invalidate the cache or use a different key for the cache

ducklake_set_sorted_multiprocess_add_column.py

The logs get printed out for process 1 before printing out process 2, but I logged out some timestamps to show the true order:

uv run ./test/ducklake_set_sorted_multiprocess_add_column.py
┌─────────┬──────────────────────┐
│ process │     sum("range")     │
│ varchar │        int128        │
├─────────┼──────────────────────┤
│ sql_1   │ 12499999997500000000 │
└─────────┴──────────────────────┘
┌─────────┬─────────────────────────────────────────────────────────────┐
│ process │ (CAST(now() AS VARCHAR) || ' sql_1 finished SET SORTED BY') │
│ varchar │                           varchar                           │
├─────────┼─────────────────────────────────────────────────────────────┤
│ sql_1   │ 2026-01-19 19:16:39.22008-07 sql_1 finished SET SORTED BY   │
└─────────┴─────────────────────────────────────────────────────────────┘


┌─────────┬─────────────────────┐
│ process │    sum("range")     │
│ varchar │       int128        │
├─────────┼─────────────────────┤
│ sql_2   │ 1999999999000000000 │
└─────────┴─────────────────────┘
┌─────────┬────────────────────────────────────────────────────────────────────┐
│ process │ (CAST(now() AS VARCHAR) || ' sql_2 finished adding column') │
│ varchar │                              varchar                               │
├─────────┼────────────────────────────────────────────────────────────────────┤
│ sql_2   │ 2026-01-19 19:16:35.416129-07 sql_2 finished adding column  │
└─────────┴────────────────────────────────────────────────────────────────────┘
┌─────────┬──────────────────────┐
│ process │     sum("range")     │
│ varchar │        int128        │
├─────────┼──────────────────────┤
│ sql_2   │ 40499999995500000000 │
└─────────┴──────────────────────┘
┌─────────┬───────────────────────────────────────────────────────┐
│ process │ (CAST(now() AS VARCHAR) || ' sql_2 about to compact') │
│ varchar │                        varchar                        │
├─────────┼───────────────────────────────────────────────────────┤
│ sql_2   │ 2026-01-19 19:16:46.373997-07 sql_2 about to compact  │
└─────────┴───────────────────────────────────────────────────────┘
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ 0 rows  │
└─────────┘
┌─────────┬───────────┬────────────┬────────────┬─────────┐
│ process │ unique_id │ sort_key_1 │ sort_key_2 │  bonus  │
│ varchar │   int64   │   int64    │  varchar   │ varchar │
├─────────┼───────────┼────────────┼────────────┼─────────┤
│ sql_2   │         3 │          1 │ woot3      │ NULL    │
│ sql_2   │         2 │          0 │ woot2      │ NULL    │
│ sql_2   │         1 │          1 │ woot1      │ NULL    │
│ sql_2   │         0 │          0 │ woot0      │ NULL    │
│ sql_2   │         7 │          1 │ woot7      │ NULL    │
│ sql_2   │         6 │          0 │ woot6      │ NULL    │
│ sql_2   │         5 │          1 │ woot5      │ NULL    │
│ sql_2   │         4 │          0 │ woot4      │ NULL    │
└─────────┴───────────┴────────────┴────────────┴─────────┘
┌─────────┬─────────────┬────────────────┬────────────────────────────────────────────┐
│ process │ snapshot_id │ schema_version │                  changes                   │
│ varchar │    int64    │     int64      │          map(varchar, varchar[])           │
├─────────┼─────────────┼────────────────┼────────────────────────────────────────────┤
│ sql_2   │           0 │              0 │ {schemas_created=[main]}                   │
│ sql_2   │           1 │              1 │ {tables_created=[main.sort_on_compaction]} │
│ sql_2   │           2 │              1 │ {tables_inserted_into=[1]}                 │
│ sql_2   │           3 │              1 │ {tables_inserted_into=[1]}                 │
│ sql_2   │           4 │              2 │ {tables_altered=[1]}                       │
│ sql_2   │           5 │              2 │ {tables_altered=[1]}                       │
│ sql_2   │           6 │              2 │ {}                                         │
└─────────┴─────────────┴────────────────┴────────────────────────────────────────────┘
┌─────────┬──────────┬────────────────┬──────────────┬────────────────┬────────────┬────────────────┬────────────┐
│ process │ table_id │ begin_snapshot │ end_snapshot │ sort_key_index │ expression │ sort_direction │ null_order │
│ varchar │  int64   │     int64      │    int64     │     int64      │  varchar   │    varchar     │  varchar   │
├─────────┼──────────┼────────────────┼──────────────┼────────────────┼────────────┼────────────────┼────────────┤
│ sql_2   │        1 │              5 │         NULL │              0 │ sort_key_1 │ DESC           │ NULLS_LAST │
│ sql_2   │        1 │              5 │         NULL │              1 │ sort_key_2 │ DESC           │ NULLS_LAST │
└─────────┴──────────┴────────────────┴──────────────┴────────────────┴────────────┴────────────────┴────────────┘